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The  impact  of  the  number  of  satellite  altimeters  used 
in  a  data  assimilative  ocean  prediction  system 
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Naval  Research  Laboratory 
Oceanography  Division 
Stennis  Space  Center,  MS  39529-5004  USA 

and 

Ole  Martin  Smedstad 
QinetiQ  North  America 
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Abstraci-This  paper  describes  the  relative  impact  of  the  number  of  satellite  altimeters  used  in  the  data  assimilation  portion  of  the  U.S. 
Navy  ocean  prediction  system.  Using  two  different  statistical  and  two  different  comparison  methods,  four  different  sets  of  results  arc 
evaluated.  Much  needed  information  is  provided,  regarding  the  ocean  prediction  accuracy  that  is  achieved  with  satellite  altimeters 
determined  relative  to  in  situ  ocean  observations. 


I.  INTRODUCTION 

During  the  years  2001  through  2003  there  were  three  separate  satellite  altimeter  data  streams  available  providing  global  sea 
surface  height  (SSHA)  measurements,  1)  JASON- 1,  2)  ERS2/Envisat,  and  3)  the  Geosat  Follow-on  (GFO).  These  systems 
provide  a  nearly  real-time  representation  of  ocean  dynamical  features  and  are  the  most  important  observation  component  of  the  U. 
S.  Navy’s  operational  global  ocean  prediction  system.  In  light  of  their  importance,  it  is  surprising  that  estimates  of  the  relative 
impact  of  the  number  of  satellite  altimeter  data  streams  on  ocean  prediction  systems  are  not  readily  available.  To  address  this 
issue,  a  main  goal  of  this  analysis  is  to  evaluate  the  relative  prediction  accuracy  resulting  from  different  numbers  and 
configurations  of  altimeters  providing  data  for  the  assimilation  portion  of  the  Navy’s  global  ocean  prediction  system. 

Satellite  altimeter  SSHA  observations  are  particularly  useful  for  mapping  eddies,  meandering  currents,  and  fronts  and  are 
assimilated  into  the  1/32°  Naval  Research  Laboratory  (NRL)  Layered  Ocean  Model  (NLOM)  [1].  NLOM  transforms  the  along 
track  data  into  high  horizontal  resolution  (eddy  resolving)  SSHA  fields  that  accurately  represent  the  mesoscale  variability.  The 
NLOM  SSHA  fields  are  then  used  by  the  Modular  Ocean  Data  Assimilation  System  (MODAS)  to  produce  three-dimensional 
fields  of  temperature  and  salinity  over  the  global  ocean.  Different  versions  of  the  MODAS  output  are  produced  from  NLOM 
fields  assimilating  data  from  different  sets  ranging  from  zero  to  three  altimeters.  These  predictions  are  then  compared  with  in  situ 
profile  observations,  which  provide  the  ground  truth  estimates. 

To  evaluate  the  accuracy  of  the  assimilation  system  relative  to  the  in  situ  ocean  observations,  we  employ  two  statistical 
methodologies.  Both  traditional  Gaussian  and  non-parametric  statistical  methods  are  used  to  ensure  that  the  results  are  not  an 
artifact  of  the  statistical  methods.  For  similar  reasons,  two  comparison  frameworks  are  used.  Comparisons  are  made  point-for- 
point  and  in  binned  overlapping  1°  radius  regions  60  days  long.  The  primary  difference  in  the  point-for-point  and  binned 
comparison  frameworks  is  that  the  binning  procedure  evens  the  statistical  weight  of  data  that  tend  to  vary  geographically 
according  to  data  density.  By  using  binned  data,  or  superposition-data,  errors  estimates  in  high  and  low  density  data  regions  have 
the  same  weight  in  global  statistics.  The  metric  for  the  comparisons  is  thermocline  depth  (TD),  since  the  SSHA  is  most  highly 
correlated  with  the  thermocline  [2].  We  also  asses  the  error  geographically,  revealing  regions  of  the  ocean  where  altimetry  has  a 
larger  impact  on  the  prediction  accuracy  than  others. 

The  result  of  applying  two  statistical  methodologies  and  two  comparison  frameworks  is  four  ways  to  view  the  relative  impact 
of  the  number  of  satellite  altimeters  used  in  an  assimilative  ocean  prediction  system.  As  a  byproduct  of  this  approach,  the  impact 
of  the  statistical  and  comparison  methods  is  also  revealed.  In  the  end,  you  will  see  that  satellite  altimetry  is  an  important 
component  of  ocean  prediction  even  when  considering  the  limitations  of  the  assimilation  system. 

H.  OBSERVATIONAL  DATA 

In  situ  ocean  profiles  of  temperature  and  salinity  from  the  World  Ocean  Database  2005  (WOD05),  the  U.  S.  Navy's  Master 
Oceanographic  Observation  Data  Set  (MOODS),  and  Argo  for  the  years  2001  through  2003  are  combined.  For  the  analysis  we 
use  only  profiles  that  have  at  least  temperature  values,  are  of  standard  quality,  have  the  shallowest  depth  level  above  12  m  and  the 
deepest  depth  level  below  150  m.  An  additional  restriction  is  that  in  the  upper  150  m  no  profile  may  have  any  gaps  in  depth 
levels  exceeding  25  m.  The  resulting  data  set  has  222,772  total  profiles  with  72,586  temperature  (T)  and  salinity  (S)  profile  pairs 
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and  150,186  T  only  profiles.  While  the  data  is  quality  controlled,  errors  including  profile  location,  XBT  drop  rate  and  other 
potential  errors  may  still  exist.  To  minimize  the  effect  of  noise  in  the  observations,  thermocline  depth  (TD),  the  approximate 
center  of  maximum  temperature  gradient  below  the  mixed  layer  is  used  for  the  evaluation.  Since  TD  is  computed  using  a  bulk 
gradient  approach,  it  is  insensitive  to  noise. 


Ill.  PREDICTION  DATA 

The  satellite  altimeter  SSHA  and  SST  data  are  directly  assimilated  into  NLOM,  an  operational,  globally  eddy-resolving 
(1/32°),  primitive  equation  6  layered  model  with  an  imbedded  bulk  mixed  layer  [1].  A  set  of  simulations  using  NLOM  were  run 
for  years  2001-2003  differing  only  in  the  number  of  altimeter  data  streams  assimilated,  from  0  to  3.  While  NLOM  has  high 
horizontal  resolution,  the  layered  formulation  means  that  the  vertical  structure  is  very  coarse.  To  achieve  better  subsurface 
predictions  we  employ  MODAS  which  estimates  profiles  of  temperature  and  salinity  from  inputs  of  SST  and/or  SSHA  using 
gridded  polynomials  at  standard  depths  with  coefficients  determined  by  least  squares  fit  to  historical  observations  [3].  For  this 
analysis,  we  input  NLOM  SSHA  and  SST  into  MODAS  to  produce  synthetic  profiles  of  temperature  and  salinity  from  which  we 
compute  the  thermocline  depth  parameters  at  all  the  observation  locations.  By  varying  the  number  of  satellite  altimeter  data 
streams  used  with  NLOM,  we  have  a  suite  of  predictions  for  comparison  with  the  observed  profiles  of  temperature  and  salinity. 
Thermocline  depth  (TD)  is  computed  from  these  predictions  at  the  time  and  location  of  each  of  the  observation  profiles  described 
in  section  11. 


IV.  COMPARISON  AND  STATISTICAL  METHODS 

Point- for-point  comparisons  are  done  by  comparing  the  observations  with  the  predictions  interpolated  to  the  location  and  time 
of  the  observations.  Statistical  error  analyses  are  then  applied  to  these  matchups.  A  second  comparison  framework  is  where 
observation  time  and  locations  are  grouped  into  analysis  windows  to  produce  superposition-data  (super-data,  hereafter).  For  this 
analysis  the  windows  are  overlapping  1°  radius  circular  regions  60  days  long  computed  each  month  for  the  entire  analysis  time 
period  from  Jan  2001  through  December  2003.  Super-data  are  defined  as  linear  least-squares  fits  to  the  model  or  observation  data 
within  the  analysis  windows  and  represent  the  mean  ao  and  temporal  trend  aj.  The  fit  parameters  ao  and  ai  are  computed  to 
produce  the  super-observations  or  super-predictions  only  for  bins  where  the  data  spans  at  least  30  of  60  days.  The  super-data 
methods  reduce  the  weight  of  individual  data  points  in  regions  of  the  ocean  with  relatively  high  observation  density.  This  is 
because  the  subsequent  statistical  analysis  treats  every  super-data  point  the  same.  As  a  result,  regions  with  a  relatively  large 
number  of  points  have  the  same  weight  as  regions  will  a  small  number  of  points.  Both  super-data  and  point-for-points  results  are 
discussed. 

Traditional  error  analysis  measure  such  as  the  mean  bias  (MB)  and  rms  error  (RMSE)  assume  Gaussian  data  distributions  that 
generally  do  not  apply  in  the  ocean.  Mean  bias  (MB)  and  RMSE,  however,  are  well  known  and  understood  quantities. 
Fortunately  there  are  non-parametric  methods  that  do  not  make  any  assumptions  about  data  distributions,  which  provide 
analogous  measures  of  the  mean  and  rms  that  are  known  in  statistical  parlance  as  location  and  scale,  respectively.  The  non- 
parametric  measures  are  theoretically  preferable  because  they  are  more  robust  when  applied  to  non-Gaussian  data  [4].  In  this 
analysis  both  Gaussian  and  non-parametric  statistical  methods  are  used  to  test  the  robustness  of  the  results. 

V.  RESULTS 

The  primary  metric  for  this  analysis  is  thermocline  depth  (TD),  which  is  computed  from  both  the  MODAS  synthetics  and  the 
observations.  Summary  diagrams  [5]  for  all  synthetics  are  presented  in  Figure  1  for  both  statistical  methodologies  and  both 
comparison  frameworks.  In  Figure  1  a  and  b  the  lowest  error  is  closest  to  the  upper  left  corner  of  the  plot  because  there  is  a 
shallow  TD  bias  (BM<0)  and  RMSE>0  for  all  synthetics.  The  statistical  methods  have  the  largest  impact  on  the  results  as  can  be 
seen  by  comparing  Figure  1  a  and  b.  The  results  using  Gaussian  statistics  have  larger  RMSE  and  standard  error  bars  than  for  non- 
parametric  statistics.  Since  Gaussian  statistics  assume  normal  data  distributions,  which  are  generally  not  valid  in  oceanographic 
data  sets,  outlier  data  points  that  violate  the  Gaussian  data  distribution  assumption  artificially  inflate  the  error  estimates.  The 
comparison  framework  has  a  substantially  smaller  impact  on  the  results.  The  only  measurable  difference  is  that  RMSE  is  larger 
for  the  point-for-point  comparison  results  (Figure  la).  The  three  altimeter  case  has  the  greatest  accuracy,  though  not  statistically 
different  from  the  two  altimeter  case  (Figure  1).  The  error  is  significantly  improved  with  the  inclusion  of  at  least  one  satellite 
altimeter  data  stream  and  three  altimeters  are  statistically  different  from  one.  The  RMSE  values  are  generally  the  same  except  for 
the  zero-altimeter  case,  which  has  the  least  accuracy. 

To  explore  the  seasonal  and  regional  error  variability,  consider  a  high  SSHA  variability  region  (HV)  bounded  by  20°  to  50°N 
and  120°E  to  160°W  and  a  low  SSHA  variability  region  (LV)  bounded  by  0°  to  20°N  and  130°E  to  150°W.  In  the  HV  region, 
the  fit  error  e  is  larger  when  TD  is  deeper  in  the  months  of  December,  January,  and  March  (compare  solid  lines  of  Figure  2,  a  and 

c).  The  %  error,  which  is  the  percent  of  the  error  relative  to  the  observed  TD  ( 100  ] )  is  related  to  the  rate  of 

change  fit  parameter  a/.  When  the  magnitude  of  Q}  is  large  the  magnitude  of  the  %  error  is  also  large  (compare  solid  black  lines 
of  Figure  2,  b  and  d).  The  %  error  tends  to  be  negative  because  the  synthetics  have  a  shallow  bias. 
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Figure  1.  In  (a)  is  the  non-parametric  statistics  summary  diagram  [5]  of  RMSE  (x-axis)  and  MB  (y-axis)  both  sealed  by  the  standard  deviation  of  the  observations. 
The  black  markers  with  75%  grey  error  bars  represent  super  data  eomparisons  while  the  50%  grey  markers  with  the  25%  grey  error  bars  are  for  the  point-for-point 
comparison  framework.  The  error  bars  represent  a  100  independent  draw  bootstrap  standard  error  estimate.  The  markers  in  the  legend  indicate  the  number  of 
satellite  altimeter  data  streams  that  were  assimilated  where  AO  represents  no  altimeters,  A1  represents  one  altimeter,  etc.  The  marker  labeled  013  is  for  three 
altimeters  used  in  an  optimal  interpolation  approach  instead  of  NLOM.  Panel  (b)  is  the  same  but  for  Gaussian  statistical  methods  and  different  axis  limits. 
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Figure  2.  The  median  fit  parameters  (a)  ao ,  (b)  ai,  (c)  e,  and  the  (d)  percent  error  and  (e)  correlation  of  ao  from  A3  synthetics  (in  black)  and  climatology  (in  grey) 
by  month,  relative  to  the  fit  parameters  determined  directly  from  observations.  The  solid  lines  are  for  the  HV  region  bounded  by  20°N  to  50°N  and  120®E  to 
160°W,  and  the  dashed-dot  lines  are  for  the  LV  region  bounded  by  0®N  to  20®N  and  130°E  to  150°W  in  the  northwest  Pacific  Ocean. 


For  the  low  variability  region,  represented  by  the  dashed-dot  lines  in  Figure  2,  the  effects  of  seasonality  are  relatively  weak  for 
Qo^  o/y  and  which  change  little  throughout  the  year.  In  panels  d  and  e  of  Figure  2,  %  error  and  correlation  coefficient  for  both 
HV  and  HL  regions  are  considered  for  both  the  three  altimeter  (A3)  synthetics  (in  black)  and  climatology  (in  grey)  cases.  In 
Figure  2,  d  and  e,  the  grey  and  black  dashed  dot  lines  are  relatively  close  together  as  compared  to  the  solid  lines.  This  suggests 
that  the  A3  and  climatology  cases  have  similar  errors  in  the  LV  region.  In  the  HV  regions,  however,  the  CLIM  case  has  larger 
magnitude  in  Figure  Id  and  lower  correlation  on  average  in  Figure  le.  The  average  correlation  coefficient  in  the  HV  region  for 
aO  determined  from  the  A3  synthetics  is  0.93,  higher  than  0.83  for  climatology.  Average  correlation  is  also  relatively  low  in  the 
LV  region,  0.84  for  both  A3  and  CLIM.  In  the  LV  region,  the  accuracy  of  the  A3  synthetics  is  not  significantly  different  from 
climatology.  The  value  added  by  altimetry  in  the  MODAS  synthetics  is  evident  only  in  the  region  of  high  SSH  variability,  where 
the  signal  stands  out  from  the  background  noise. 

VI.  SUMMARY  AND  CONCLUSIONS 

During  the  analysis  time  of  this  study  (2001-2003)  there  were  three  satellite  altimeter  data  streams  available  in  near-real  time 
for  operational  ocean  prediction.  Surprisingly,  the  relative  accuracy  of  operational  ocean  predictions  afforded  by  these  data 
streams  is  not  readily  available.  To  address  this  issue,  this  paper  investigates  the  relative  prediction  accuracy  resulting  from 
different  numbers  and  configurations  of  altimeters  providing  data  for  the  assimilation  portion  of  the  Navy’s  global  ocean 
prediction  system. 

The  general  results  show  that  accuracy  provided  by  the  satellite  altimeter  data  streams  is  greater  in  regions  of  high  SSHA 
variability,  and  significant  error  reduction  is  achieved  with  the  addition  of  at  least  one  satellite  altimeter  data  stream.  Additional 
error  reduction  when  assimilating  data  from  three  altimeters  versus  one  is  significant  only  in  bias.  These  conclusions  are  drawn 
from  both  the  point-for-point  and  super-data  frameworks  when  using  non-parametric  statistical  methods.  The  lack  of  significant 
skill  improvement  between  two  and  three  satellite  altimeter  data  sets  may  be  a  consequence  of  the  length  and  time  scales 
associated  with  the  NLOM  assimilation  and  the  smoothing  associated  with  gridded  climatological  coefficients  of  MODAS.  As  a 
result,  this  system  is  unable  to  take  full  advantage  of  the  added  spatial  detail  provided  by  multiple  altimeters.  The  simulated  error 
experiment  of  Smedtad  et  al.  [6]  also  indicates  that  the  marginal  reduction  in  SSHA  error  decreases  for  each  additional  satellite 
data  stream. 

The  statistical  methodologies  are  found  to  have  a  relatively  large  impact  on  the  results.  Evaluation  based  on  traditional 
statistics  that  assume  Gaussian  data  distributions  result  in  inflated  error  and  error  uncertainty  estimates.  Using  non-parametric 
statistics  circumvents  the  need  for  strict  and  often  unfounded  assumptions  about  the  data  distribution.  The  two  comparison 
frameworks  used  in  this  analysis  have  a  lesser  impact.  The  effect  of  the  super-data  methods  is  to  even  the  influence  of  data 
between  low  and  high  density  observation  regions.  Since  the  global  error  results  for  the  point-for-point  framework  are  slightly 
larger  than  for  the  super-data  framework,  high  density  observation  regions  tend  to  occur  in  areas  with  larger  errors. 

Comparing  predictions  to  in  situ  observations  is  the  ultimate  test  in  that  the  observations  are  the  closest  information  we  have  to 
the  true  ocean  condition.  This  type  of  comparison,  however,  is  inherently  plagued  with  representation  errors  because  the 
observations  generally  have  more  physical  processes  influencing  the  measurements  on  shorter  space  and  time  scales  than  the 
predictions  are  able  to  resolve. 
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